Modifying robot dynamics in response to human presence

ABSTRACT

A robot system models the behavior of a user when the user occupies an operating zone associated with a robot. The robot system predicts future behaviors of the user, and then determines whether those predicted behaviors interfere with anticipated behaviors of the robot. When such interference may occur, the robot system generates dynamics adjustments that can be implemented by the robot to avoid such interference. The robot system may also generate dynamics adjustments that can be implemented by the user to avoid such interference.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention relate generally to robotics and,more specifically, to modifying robot dynamics in response to humanpresence.

Description of the Related Art

In a conventional manufacturing environment, an industrial robotoperates within a three-dimensional (3D) operating zone. The operatingzone associated with a given robot is defined by the mechanical limitsof the robot and/or the specific actions the robot is programmed toperform. For example, the operating zone associated with a 1-axis drillpress robot could be limited to a cylinder that is aligned with the axisand within which the robot is capable of operating. By contrast, theoperating zone associated with a 6-axis welding robot could include asphere that surrounds the robot and defines a 3D area within which therobot is capable of operating.

The operating zones associated with industrial robots are usually unsafeareas for humans to occupy. In particular, industrial robots areoftentimes quite large and can move in rapid and unpredictable manners,potentially causing harm to nearby humans or human operators. Despitethese known risks, in industrial settings, humans usually need to enterthe operating zones of robots for various reasons. For example, a repairtechnician may need to enter the operating zone of a robot in order todiagnose and/or fix a problem with the robot. Alternatively, a designengineer may need to enter the operating zone of a robot to inspect aproduct or part the robot is fabricating. When a human does enter theoperating zone of a robot, the robot is typically powered down to allowthe human to enter the operating zone. However, this approach suffersfrom specific drawbacks.

First, the human cannot directly observe the actions of the robot whenthe robot is powered down. This limitation makes diagnosing problemsdifficult and also prevents direct observation of how the robot performsvarious operations. Second, the human cannot interact with the robot inany meaningful way, thereby limiting the degree to which a human canimprove, for example, the way a robot performs a given fabricationprocess.

As the foregoing illustrates, what is needed in the art are moreeffective techniques for interacting with industrial robots.

SUMMARY OF THE INVENTION

Various embodiments of the present invention set forth acomputer-implemented method for controlling a robot, including capturingsensor data related to an operating zone within which the robot performsa fabrication process, generating a dynamic model associated with a userwho is residing within the operating zone, predicting a first useraction associated with the user based on the dynamic model, andadjusting one or more dynamics of the robot based on the first useraction.

At least one advantage of the techniques described herein is that theuser may occupy the operating zone associated with the robot withreduced risk.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 illustrates a system configured to implement one or more aspectsof the present invention;

FIG. 2 illustrates an operating zone surrounding the robot of FIG. 1that is transected by different paths associated with a user, accordingto various embodiments of the present invention;

FIG. 3 illustrates in greater detail the control application of FIG. 1,according to various embodiments of the present invention;

FIG. 4 is a flow diagram of method steps for modifying the dynamics of arobot in response to the presence of a user, according to variousembodiments of the present invention;

FIGS. 5A-5B illustrate how the control application of FIG. 1 adjusts thedynamics of the robot of FIG. 1 in response to the presence of a user,according to various embodiments of the present invention;

FIGS. 6A-6B illustrate how the control application of FIG. 1 adjusts theconfiguration of the robot of FIG. 1 in response to the presence of auser, according to various embodiments of the present invention;

FIGS. 7A-7B illustrate how the control application of FIG. 1 constrainsthe dynamics of the robot of FIG. 1 based on the location of a user,according to various embodiments of the present invention; and

FIG. 8A-8B illustrate how the robot system of FIG. 1 interacts with auser to optimize a fabrication process, according to various embodimentsof the present invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the present invention. However,it will be apparent to one of skill in the art that the presentinvention may be practiced without one or more of these specificdetails.

As discussed above, in a conventional manufacturing environment,industrial robots operate within specific operating zones that may bedangerous for humans to occupy. For various reasons, humans must enterthese zones anyway. In the interest of safety, industrial robots aretypically powered down in the presence of humans. However, doing solimits the effectiveness with which humans can interact with thoserobots.

To address these issues, embodiments of the invention include a robotsystem configured to detect nearby human users and then adjust thedynamics of a robot based on the predicted behavior of those users.Accordingly, a user may safely enter the operating zone of the robot andinteract with the robot.

System Overview

FIG. 1 illustrates a system configured to implement one or more aspectsof the present invention. As shown, a robot system 100 includes a robot110 coupled to a computing device 120. Robot 110 is configured tofabricate a structure 130 in the presence of a user 140. Structure 130may be any technically feasible three-dimensional (3D) structure,including an assembly of polygons, among other possibilities.

Robot 110 includes a sensor array 112 configured to capturethree-dimensional optical and/or acoustic data related to structure 130,user 140, and/or the environment surrounding robot 110. For example,sensor array 112 could include a set of stereoscopic cameras configuredto collect stereoscopic images representing the spatial environmentsurrounding robot 110. Those images may represent nearby objects such asstructure 130 and/or user 140. In addition, sensor array 112 couldinclude a set of acoustic sensors configured to measure the acousticsoundscape surrounding robot 110. That soundscape could include soundsgenerated via fabrication of structure 130 and/or sounds generated byuser 140. The optical and/or acoustic data gathered by sensor array 112is referred to generally as “audiovisual data.”

Computing device 120 collects audiovisual data via sensor array 112 andthen adjusts the dynamics of robot 110 based on the presence andpredicted behavior of user 140. Computing device 120 includes aprocessor 122, input/output (I/O) devices 124, and a memory 126.Processor 122 may be any technically feasible form of processing deviceconfigured to process data and execute program code. Processor 122 couldbe, for example, a central processing unit (CPU), a graphics processingunit (GPU), an application-specific integrated circuit (ASIC), afield-programmable gate array (FPGA), any technically feasiblecombination of such units, and so forth.

I/O devices 124 may include devices configured to receive input,including, for example, a keyboard, a mouse, and so forth. I/O devices124 may also include devices configured to provide output, including,for example, a display device, a speaker, and so forth. I/O devices 124may further include devices configured to both receive and provide inputand output, respectively, including, for example, a touchscreen, auniversal serial bus (USB) port, and so forth.

Memory 126 may include any technically feasible storage mediumconfigured to store data and software applications. Memory 126 could be,for example, a hard disk, a random access memory (RAM) module, or aread-only memory (ROM), among others. Memory 126 includes a controlapplication 128 and a database 130. Control application 128 is asoftware application that, when executed by processor 122, adjusts thedynamics of robot 110 in response to user 140 occupying an operatingzone associated with robot 110. The operating zone of robot 110 is shownin FIG. 2.

FIG. 2 illustrates an operating zone surrounding the robot of FIG. 1that is transected by different paths associated with a user, accordingto various embodiments of the present invention. As shown, operatingzone 200 surrounds robot 110. Operating zone 200 may reflect themechanical limits of robot 110 or a specific area within which robot 110is programmed to operate. Operating zone 200 also includes variousfabrication elements robot 110 may use when fabricating structure 130,including implements 202 and materials 204.

As also shown, user 140 traverses operating zone 200 along variouspathways in order to perform different tasks. For example, user 140could traverse path 210(A) to collect materials 204. Similarly, user 140could traverse path 200(B) to inspect structure 130. User 140 could alsotraverse path 200(C) to exit operating zone 200. Via sensor array 112,robot system 100 is configured to collect audiovisual data thatrepresents the behavior of user 140. That behavior may include thetraversal of paths 210, as discussed, as well as any other behaviorperformed by user 140. Based on that audiovisual data, controlapplication 128 modifies the dynamics of robot 110 to accommodate thepresence of user 140, thereby creating a safe environment within whichuser 140 can interact with robot 110. Control application 128 isdiscussed below in conjunction with FIG. 3.

Software Overview

FIG. 3 illustrates in greater detail the control application of FIG. 1,according to various embodiments of the present invention. As shown,control application 128 includes a spatial analyzer 300, a dynamicsadjuster 310, a natural language (NL) generator 320, and a commandgenerator 330.

Spatial analyzer 300 is a software module configured to model thedynamics of user 140 based on audiovisual data 302 to generate usermodel 312. In doing so, spatial analyzer 300 may implement computervision techniques or other machine learning approaches. For example,spatial analyzer 300 could implement a spatial mapping approach togenerate a point cloud representation of user 140 at a given time.Spatial analyzer 300 could repeat this process to identify cohesiveportions of this point cloud that may correspond to the body, legs,arms, and so forth of user 140. Spatial analyzer 300 would then analyzeand predict the motion of these cohesive portions. Alternatively,spatial analyzer 300 could model the dynamics of user 140 via motioncapture techniques based on markers coupled to user 140. Spatialanalyzer 300 transmits user model 312 to dynamics adjuster 310.

Dynamics adjuster 310 is a software module configured to adjust thedynamics of robot 110 based on the dynamics of user 140 set forth inuser model 312. Dynamics adjuster 310 analyzes user model 312 to predictthe position and configuration (or pose) of user 140 at various futuretimes. Dynamics adjuster 310 also analyzes fabrication program 314 todetermine the planned position and configuration of robot 110 at thosefuture times.

Fabrication program 314 is a baseline program that establishes bothhigh-level objectives for robot 110 to accomplish during fabrication ofstructure 130, and different sets of discrete tasks robot 110 mayperform to achieve those objectives. In one embodiment, fabricationprogram 314 may include specific numerical control (NC) programs thatrobot 110 executes to perform such discrete tasks. In anotherembodiment, dynamics adjuster 310 determines the planned position andconfiguration of robot 110 by simulating the execution of fabricationprogram 314.

Based on user model 312 and fabrication program 314, dynamics adjuster310 generates robot dynamics adjustments 322. Robot dynamics adjustments322 include modifications to the current dynamics of robot 110 thatcause robot 110 to accommodate the presence of user 140 at the aforesaidfuture times. Robot dynamics adjustments 322 could, for example, causerobot 110 to move more slowly as user 140 approaches robot 110.Alternatively, robot dynamics adjustments 322 could cause robot 110 toassume a different configuration in order to avoid a predicted collisionwith user 140 during operation. Robot dynamics adjustments 322 couldalso constrain robot to prevent the initiation of specific tasks thatwould be dangerous for user 140, such as arc welding in the presence ofuser 140.

In one embodiment, dynamics adjuster 310 extrapolates the dynamics ofuser 140 based on user model 312 to predict the position andconfiguration of user 140 at a specific time in the future. Dynamicsadjuster 310, could for example, extrapolate the movements of cohesivegroups of points associated with a point cloud representation of user140, such as that discussed above. Then, dynamic adjuster 310 woulddetermine a future position and configuration of user 140 at a futuretime. Dynamics adjuster 310 could also map this position andconfiguration to an identifiable action, such as “walking”, “holding”,and so forth. Dynamics adjuster 310 compares the predicted position andconfiguration (or action) of user 140 at the specific time to theplanned configuration of robot 110 derived from fabrication program 314.Based on this comparison, dynamics adjuster 310 determines whether acollision, or near collision, could potentially occur between user 140and robot 110 at the specific time. Dynamics adjuster 310 may performthis procedure for any range of future times to identify one or manypotential collisions. Then, dynamics adjuster 310 generates robotdynamics adjustments 322 that cause robot 110 to avoid the predictedcollision(s).

Command generator 320 receives robot dynamics adjustments 322 and thengenerates commands 340. Robot 110 executes commands 340 to implementmodified dynamics that accommodate the predicted behavior of user 140.This approach may provide a safer working environment within which bothrobot 110 and user 140 can cooperate.

In addition to generating robot dynamics adjustments 322, dynamicsadjuster 310 may also generate user dynamics adjustments 332 based onuser model 312 and fabrication program 314. User dynamics adjustments332 represent changes to the behavior of user 140 that may protect user140 from danger or enhance cooperation between user 140 and robot 110.For example, user dynamics adjustments 332 could specify a position thatwould be objectively safer (e.g., would avoid collisions) for user 140to occupy during a particular operation of robot 110. Alternatively,user dynamics adjustments 332 could indicate a particular action thatuser 140 could perform to optimize a fabrication task performed by robot110 and user 140 cooperatively.

NL generator 330 processes user dynamics adjustments 332 and generatessuggestions 350. Suggestions 350 include human language that can becommunicated to user 140 and that describe user dynamics adjustments332. I/O devices 124 output suggestions 350 as text or audio, amongother forms of communication. User 140 may then perform the suggestedactions.

Robot system 100 may implement the techniques discussed thus far topromote user safety within operating zone 200. Robot system 100 may alsoimplement these techniques to promote cooperation with user 140. FIG. 4describes these techniques in a procedural manner, and FIGS. 5A-8Bdescribed various examples of how robot 110 and user 140 safelyco-occupy operating zone 200.

FIG. 4 is a flow diagram of method steps for modifying the dynamics of arobot in response to the presence of a user, according to variousembodiments of the present invention. Although the method steps aredescribed with respect to the systems of FIGS. 1-3, persons skilled inthe art will understand that any system configured to perform the methodsteps, in any order, falls within the scope of the present disclosure.

As shown, a method 400 begins at step 402, where control application 128causes robot to perform operations associated with fabrication ofstructure 130. Control application 128 could, for example, executefabrication program 314. At step 404, control application 128 capturesaudiovisual data 302 from within operating zone 200 surrounding robot110. At step 406, control application 128 identifies user 140 performingone or more actions within operating zone 200 based on audiovisual data302.

At step 408, control application 128 generates a prediction of actionsperformed by user 140. Control application 128 could generate thisprediction based on user model 312. At step 410, control application 128generates robot dynamics adjustments 322 based on the predicted actionsof user 140. Robot dynamics adjustments 322 generally cause robot 110 toaccommodate the presence of user 140 and/or cooperate directly with user140 when implemented. At step 412, control application 128 modifiesfabrication operations performed by robot 110 based on robot dynamicsadjustments 322 generated at step 412.

In addition, at step 414, control application 128 generates suggestedmodifications to the actions performed by user 140 based on theprediction of those actions generated at step 408. At step 416, controlapplication 128 outputs these suggested modifications to user 140. Steps414 and 416 may enhance interactions between robot 110 and user 140, andpotentially allow those interactions to evolve over time. For example,as robot 110 provides suggestions to user 140, user 140 may in turnconceptualize improvements to the fabrication process performed by robot110, and could then implement those improvements. In response, robot 110could identify additional adjustments user 140 could make, and providecorresponding suggestions.

In this manner, control application 128 may preserve the safety of user140 while also allowing robot 110 to continue fabrication operations.Further, control application 128 facilitates safe collaboration betweenuser 140 and robot 110. The general techniques described thus far arealso discussed below in conjunction with various examples shown in FIGS.5A-8B.

Exemplary Interactions Between Robot and User

FIGS. 5A-5B illustrate how the control application of FIG. 1 adjusts thedynamics of the robot of FIG. 1 in response to the presence of a user,according to various embodiments of the present invention.

As shown in FIG. 5A, user 140 traverses path 210(A) towards materials204 while robot 110 performs a fabrication maneuver 500. Fabricationmaneuver 500 involves rotating robot 110 at a high rotation rate. Underordinary circumstances, this maneuver may be dangerous to user 140.However, control application 128 is configured to mitigate that danger,as discussed below.

As shown in FIG. 5B, control application 128 causes robot 110 to slowdown and perform fabrication maneuver 510 instead of fabricationmaneuver 500. Control application 128 implements fabrication maneuver510 in response to the presence and/or predicted behavior of user 140.In particular, control application 128 gathers audiovisual data 302 andthen models the dynamics of user 140, thereby predicting that user 140will traverse path 210(A). Since path 210(A) approaches robot 110,control application 128 adjusts the dynamics of robot 110 accordingly.

Because fabrication maneuver 510 is slower than fabrication maneuver500, user 140 may have sufficient time to predict the motions of robot110 and potentially avoid danger. In one embodiment, control application128 reduces the speed of rotation associated with robot 110 inproportion to the distance between robot 110 and user 140. Controlapplication 128 may also reconfigure robot 110 based on the positionand/or configuration/pose of user 140, as described in greater detailbelow in conjunction with FIGS. 6A-6B.

FIGS. 6A-6B illustrate how the control application of FIG. 1 adjusts theconfiguration of the robot of FIG. 1 in response to the presence of auser, according to various embodiments of the present invention.

As shown in FIG. 6A, user 140 traverses path 510(C) close to where robot110 performs fabrication operations to fabricate structure 130. Controlapplication 128 configures robot 110 according to configuration 600.Configuration 600 includes a specific set of joint angles according towhich robot 110 is articulated. In configuration 600, robot 110 mayblock user 140 as user 140 traverses path 210(C), which may be dangerousto user 140 in a conventional setting. However, control application 128is configured to mitigate this danger, as described below in conjunctionwith FIG. 6B.

As shown in FIG. 6B, control application 128 causes robot 110 to assumeconfiguration 610, thereby accommodating the presence of user 140 asuser 140 traverses path 210(C). Control application 128 gathersaudiovisual data 302 and models the dynamics of user 140, then predictsthat user 140 may collide with robot 110 when traversing path 210(C). Inresponse, control application 128 adjusts the configuration of robot 110to permit user 140 to safely pass under robot 110 and traverse path210(C).

In various embodiments, control application 128 implements a geneticalgorithm to determine one or more possible configurations for robot 110that do not result in a collision with user 140. Then, controlapplication 128 causes robot 110 to assume one such configuration.Control application 128 may also cause robot 110 to continue fabricationof structure 130 once reconfigured. With this approach, user 140 maysafely occupy operating zone 200 while robot 110 fabricates structure130. Control application 128 may also constrain the dynamics of robot110 relative to the presence and/or predicted behavior of user 140, asdescribed in greater detail below in conjunction with FIGS. 7A-7B.

FIGS. 7A-7B illustrate how the control application of FIG. 1 constrainsthe dynamics of the robot of FIG. 1 based on the location of a user,according to various embodiments of the present invention. As shown inFIGS. 7A-7B, operating zone 200 is divided into subzones 700(0) through700(11). User 140 may occupy any of subzones 700. Based on theparticular subzone 700 user 140 occupies, control application 128generates a specific set of dynamic operations available to robot 110.

For example, as shown in FIG. 7A, when user 140 occupies subzone 700(8),control application 128 generates set 710(0) of dynamics operations. Set710(0) includes operations 712(0), 712(1), 712(4), and 712(7). Inanother example, as shown in FIG. 7B, when user 140 occupies subzone700(4), control application 128 generates set 710(1) of dynamicsoperations. Set 710(1) includes operations 712(1), 712(4), and 712(5).

Control application 128 determines which dynamics operations 712 shouldbe allowed when user 140 occupies a given subzone 700 based on whetherthose operations could potentially cause harm to user 140. Specifically,control application 128 determines which operations may endanger user140 (deemed “unsafe” operations) and which operations may not (deemed“safe” operations). Control application 128 then includes the “safe”operations in the set 710 associated with the given subzone 700 and doesnot include the “unsafe” operations. In one embodiment, controlapplication 128 is configured to recognize “dangerous” or “unsafe”operations, which could include, for example, collisions between robot110 and user 140, among others. In another embodiment, controlapplication 128 may also identify specific combinations of operationsthat may endanger user 140, and prevents all such operations from beingincluded together in a given set 710. In yet another embodiment, controlapplication 128 performs the above approach based on a prediction of thesubzone 700 user 140 may occupy in the future.

Referring generally to FIGS. 5A-7B, the various techniques discussed byway of example thus far are generally meant to preserve the safety ofuser 140 when user 140 occupies operating zone 200. In addition,however, control application 128 may perform certain operations thatfacilitate interaction between user 140 and robot 110, as describedbelow in conjunction with FIGS. 8A-8B.

FIGS. 8A-8B illustrate how the robot of FIG. 1 interacts with a user tooptimize a fabrication process, according to various embodiments of thepresent invention.

As shown in FIG. 8A, user 140 stands proximate to robot 110 and holds astructural element 800 at a height 802. Structural element 800 can beused to fabricate structure 130. User 140 attempts to provide structuralelement 800 to robot 110 to aid in the fabrication process. However,robot 110 occupies configuration 810 and cannot reach structural element800. Control application 128 recognizes that fabrication is impededbecause robot 110 cannot acquire structural element 800. In response,control application 128 determines specific dynamics for user 140 toimplement that may facilitate fabrication.

Specifically, as shown in FIG. 8B, control application 128 generatessuggestion 820 indicating that user 140 should raise structural element800 one foot higher to height 804. Thus positioned, robot 110 may assumeconfiguration 812 and obtain structural element 800 from user 140. Inthis manner, control application 128 allows robot 110 and user 140 tocooperate to perform fabrication tasks. Persons skilled in the art willunderstand that control application 128 may generate suggestionsassociated with any technically feasible fabrication task, and may alsogenerate suggestions meant to preserve the safety of user 140. Forexample, control application 128 could instruct user 140 to move to aspecific subzone 700 to avoid a dangerous fabrication operation.

Control application 128 implements any of the techniques described thusfar, and any combination thereof, to control robot 110 when user 140occupies operating zone 200. These techniques promote the safety of user140 and also permit robot-human collaboration.

In sum, a robot system models the behavior of a user when the useroccupies an operating zone associated with a robot. The robot systempredicts future behaviors of the user, and then determines whether thosepredicted behaviors interfere with anticipated behaviors of the robot.When such interference potentially occurs, the robot system generatesdynamics adjustments that can be implemented by the robot to avoid suchinterference. The robot system may also generate dynamics adjustmentsthat can be implemented by the user to avoid such interference.

At least one advantage of the techniques described herein is that theuser may occupy the operating zone associated with the robot withreduced risk. Further, the user can perform a variety of actions, andthe robot adaptively adjusts dynamics to accommodate those actions. Inaddition, the user and robot may interact with one another in a moreeffective manner, thereby improving a fabrication processes that involveboth humans and robots.

1. Some embodiments of the invention include a computer-implementedmethod for controlling a robot, the method comprising: capturing sensordata related to an operating zone within which the robot performs afabrication process, generating a dynamic model associated with a userwho is residing within the operating zone, predicting a first useraction associated with the user based on the dynamic model, andadjusting one or more dynamics of the robot based on the first useraction.

2. The computer-implemented method of clause 1, wherein the sensor datacomprises a point cloud representation of the operating zone, andgenerating the dynamic model comprises analyzing the point cloudrepresentation to identify one or more sets of points that move togetherover time.

3. The computer-implemented method of any of clauses 1 and 2, whereinpredicting the first user action comprises: determining a first movementassociated with the one or more sets of points, and mapping the firstmovement to the first user action.

4. The computer-implemented method of any of clauses 1, 2, and 3,wherein predicting the first user action comprises predicting a firsttime at which the first user action will occur.

5. The computer-implemented method of any of clauses 1, 2, 3, and 4,wherein adjusting one or more dynamics of the robot comprises: analyzinga fabrication program to identify a first robot action that isprogrammed to occur at the first time, identifying a conflict betweenthe first user action and the first robot action, causing the robot toperform a second robot action at the first time instead of the firstrobot action to avoid the conflict.

6. The computer-implemented method of any of clauses 1, 2, 3, 4, and 5,wherein identifying the conflict comprises determining that the robotwill collide with the user at the first time when the robot performs thefirst robot action and the user performs the first user action.

7. The computer-implemented method of any of clauses 1, 2, 3, 4, 5, and6, wherein adjusting one or more dynamics of the robot comprises causingthe robot to reduce at least one speed associated with a fabricationprocess.

8. The computer-implemented method of any of clauses 1, 2, 3, 4, 5, 6,and 7, wherein adjusting one or more dynamics of the robot comprisescausing the robot to reorient at least one joint when performing afabrication process.

9. The computer-implemented method of any of clauses 1, 2, 3, 4, 5, 6,7, and 8, further comprising: processing the sensor data to identify afirst subzone within the operating zone, and generating a set ofdynamics operations corresponding to the first subzone, wherein therobot is able to perform any dynamics operation included in the set ofdynamics operations so long as the user resides within the firstsubzone.

10. Some embodiments of the invention include a non-transitorycomputer-readable medium storing program instructions that, whenexecuted by a processor, cause the processor to control a robot byperforming the steps of: capturing sensor data related to an operatingzone within which the robot performs a fabrication process, generating adynamic model associated with a user who is residing within theoperating zone, predicting a first user action associated with the userbased on the dynamic model, and adjusting one or more dynamics of therobot based on the first user action.

11. The non-transitory computer-readable of clause 10, wherein thesensor data comprises a point cloud representation of the operatingzone, and the step of generating the dynamic model comprises analyzingthe point cloud representation to identify one or more sets of pointsthat move together over time.

12. The non-transitory computer-readable medium of any of clauses 10 and11, wherein the step of predicting the first user action comprises:determining a first movement associated with the one or more sets ofpoints, and mapping the first movement to the first user action.

13. The non-transitory computer-readable of any of clauses 10, 11, and12, wherein the step of predicting the first user action comprisespredicting a first time at which the first user action will occur, andwherein the step of adjusting one or more dynamics of the robotcomprises: analyzing a fabrication program to identify a first robotaction that is programmed to occur at the first time, identifying aconflict between the first user action and the first robot action,causing the robot to perform a second robot action at the first timeinstead of the first robot action to avoid the conflict.

14. The non-transitory computer-readable of any of clauses 10, 11, 12,and 13, wherein identifying the conflict comprises determining that therobot will collide with the user at the first time when the robotperforms the first robot action and the user performs the first useraction.

15. The non-transitory computer-readable of any of clauses 10, 11, 12,13, and 14, further comprising the steps of: processing the sensor datato identify a first subzone within the operating zone, generating a setof dynamics operations corresponding to the first subzone, wherein therobot is able to perform any dynamics operation included in the set ofdynamics operations so long as the user resides within the firstsubzone.

16. The non-transitory computer-readable of any of clauses 10, 11, 12,13, 14, and 15, wherein the robot is not able to perform any dynamicsoperations other than the dynamics operations included in the set ofdynamics operations while the user resides within the first subzone.

17. The non-transitory computer-readable of any of clauses 10, 11, 12,13, 14, 15, and 16, further comprising the steps of: analyzing afabrication program to identify a first robot action that is to occur ata later time, determining a second user action that facilitates thefirst robot action, indicating the second user action to the user.

18. The non-transitory computer-readable of any of clauses 10, 11, 12,13, 14, 15, 16, and 17, wherein the step of indicating the second useraction to the user comprises: generating a natural language descriptionof the second user action, and outputting the natural languagedescription to the user.

19. Some embodiments of the invention include a system, comprising: arobot that performs a fabrication process within an operating zone, asensor array that captures sensor data related to the operating zone, amemory storing a control application, and a processor that executes thecontrol application to perform the steps of: generating a dynamic modelassociated with a user who is residing within the operating zone,predicting a first user action associated with the user based on thedynamic model, and adjusting one or more dynamics of the robot based onthe first user action.

20. The system of clause 1, wherein the robot comprises a jointed arm,and wherein the processor adjusts the one or more dynamics of the robotby modifying at least one joint angle associated with the robot.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present invention andprotection.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “module” or“system.” Furthermore, aspects of the present disclosure may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, enable the implementation of the functions/acts specified inthe flowchart and/or block diagram block or blocks. Such processors maybe, without limitation, general purpose processors, special-purposeprocessors, application-specific processors, or field-programmableprocessors or gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

The invention claimed is:
 1. A computer-implemented method forcontrolling a robot, the method comprising: capturing sensor datarelated to an operating zone within which the robot performs afabrication process; generating a dynamic model associated with a userwho is residing within the operating zone; predicting, based on thedynamic model, a predicted user action that the user is to perform at afirst later time; generating, based on the predicted user action, a setof dynamics adjustments of the robot for performing a first robot actionin the fabrication process; determining, based on the predicted useraction, a suggested user action in the operating zone that facilitatesthe first robot action; and generating a first suggestion indicating thesuggested user action, wherein an output device provides the firstsuggestion to the user.
 2. The computer-implemented method of claim 1,wherein the sensor data comprises a point cloud representation of theoperating zone, and generating the dynamic model comprises analyzing thepoint cloud representation to identify one or more sets of points thatmove together over time.
 3. The computer-implemented method of claim 2,wherein predicting the predicted user action comprises: determining afirst movement associated with the one or more sets of points; andmapping the first movement to the predicted user action.
 4. Thecomputer-implemented method of claim 1, wherein predicting the predicteduser action comprises determining the first later time at which thepredicted user action will occur.
 5. The computer-implemented method ofclaim 4, wherein generating the set of dynamics adjustments comprises:analyzing a fabrication program to identify a first robot action that isprogrammed to occur at the first later time; identifying a conflictbetween the predicted user action and the first robot action; causingthe robot to perform a second robot action at the first later timeinstead of the first robot action in order to avoid the conflict.
 6. Thecomputer-implemented method of claim 5, wherein identifying the conflictcomprises determining that the robot will collide with the user at thefirst later time when the robot performs the first robot action and theuser performs the predicted user action.
 7. The computer-implementedmethod of claim 1, wherein generating the set of dynamics adjustmentscomprises causing the robot to reduce at least one speed associated withthe fabrication process.
 8. The computer-implemented method of claim 1,wherein generating the set of dynamics adjustments comprises causing therobot to reorient at least one joint when performing the fabricationprocess.
 9. The computer-implemented method of claim 1, furthercomprising: processing the sensor data to identify a first subzonewithin the operating zone; and generating a set of dynamics operationscorresponding to the first subzone, wherein the robot is able to performany dynamics operation included in the set of dynamics operations solong as the user resides within the first subzone.
 10. One or morenon-transitory computer-readable media storing program instructionsthat, when executed by one or more processors, cause the one or moreprocessors to control a robot by performing the steps of: capturingsensor data related to an operating zone within which the robot performsa fabrication process; generating a dynamic model associated with a userwho is residing within the operating zone; predicting, based on thedynamic model, a predicted user action that the user is to perform at afirst later time; generating, based on the predicted user action, a setof dynamics adjustments of the robot for performing a first robot actionin the fabrication process determining, based on the predicted useraction, a suggested user action in the operating zone that facilitatesthe first robot action; and generating a first suggestion indicating thesuggested user action, wherein an output device provides the firstsuggestion to the user.
 11. The one or more non-transitorycomputer-readable media of claim 10, wherein the sensor data comprises apoint cloud representation of the operating zone, and the step ofgenerating the dynamic model comprises analyzing the point cloudrepresentation to identify one or more sets of points that move togetherover time.
 12. The one or more non-transitory computer-readable media ofclaim 11, wherein the step of predicting the predicted user actioncomprises: determining a first movement associated with the one or moresets of points; and mapping the first movement to the predicted useraction.
 13. The one or more non-transitory computer-readable media ofclaim 10, wherein: the step of predicting the predicted user actioncomprises determining the first later time at which the predicted useraction will occur, and the step of generating the set of dynamicsadjustments comprises: analyzing a fabrication program to identify afirst robot action that is programmed to occur at the first later time;identifying a conflict between the predicted user action and the firstrobot action; causing the robot to perform a second robot action at thefirst later time instead of the first robot action in order to avoid theconflict.
 14. The one or more non-transitory computer-readable media ofclaim 13, wherein identifying the conflict comprises determining thatthe robot will collide with the user at the first later time when therobot performs the first robot action and the user performs thepredicted user action.
 15. The one or more non-transitorycomputer-readable media of claim 10, further comprising the steps of:processing the sensor data to identify a first subzone within theoperating zone; and generating a set of dynamics operationscorresponding to the first subzone, wherein the robot is able to performany dynamics operation included in the set of dynamics operations solong as the user resides within the first subzone.
 16. The one or morenon-transitory computer-readable media of claim 15, wherein the robot isnot able to perform any dynamics operations other than the dynamicsoperations included in the set of dynamics operations while the userresides within the first subzone.
 17. The one or more non-transitorycomputer-readable media of claim 10, wherein: the step of generating thefirst suggestion indicating the suggested user action comprisesgenerating a natural language description of the first suggestion; andfurther comprising outputting, to the user, the natural languagedescription the first suggestion.
 18. A system, comprising: a robot thatperforms a fabrication process within an operating zone; a sensor arraythat captures sensor data related to the operating zone; a memorystoring a control application; and a processor that executes the controlapplication to perform the steps of: generating a dynamic modelassociated with a user who is residing within the operating zone,predicting, based on the dynamic model, a predicted user action that theuser is to perform at a first time, generating, based on the predicteduser action, a set of dynamics adjustments of the robot for performing afirst robot action in the fabrication process, determining, based on thepredicted user action, a suggested user action in the operating zonethat facilitates the robot action, and generating a first suggestionindicating the suggested user action, wherein an output device providesthe first suggestion to the user.
 19. The system of claim 18, wherein:the robot comprises a jointed arm, and the processor further executesthe control application to generate the set of dynamics adjustments bymodifying at least one joint angle associated with the robot.
 20. Thecomputer-implemented method of claim 1, wherein the set of dynamicsadjustments are generated based on a predicted pose associated with theuser.
 21. The computer-implemented method of claim 1, wherein afabrication program comprises numerical control programs that the robotexecutes for performing the fabrication process.
 22. Thecomputer-implemented method of claim 21, wherein generating the set ofdynamics adjustments comprises analyzing the fabrication program toidentify a planned position and configuration of the robot at the firstlater time.